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IMPACT OF DATA ANALYTICS 

The most dramatic factor reshaping the future of organizations and busi¬ 
ness is data analytics, which offers a new model for decision making and 
understanding the world around us. Basing decisions on data seems an 
obvious way for organizations to improve output and productivity, but for 
higher education, experience and gut instinct continue to dominate policy 
and decision making, often causing barriers to educational reform. 

Outside of higher education, the range of data-driven practices is exten¬ 
sive. In Moneyball, Michael Lewis chronicled how Billy Beane, the manager 
of the low-budget Oakland Athletics, massaged data and arcane baseball 
statistics to spot undervalued players. Retailers like Wal-Mart, Kohl's, and 
Target analyze sales, pricing, and demographics to tailor product selections 
in stores and determine the timing of markdowns. The shipping company 
UPS mines data on truck delivery times and traffic patterns to fine-tune rout¬ 
ing. Online dating services like Match.com use client personal characteristics 
to improve algorithms for matching men and women. Police departments 
use computerized mapping and analysis of variables like arrest patterns. 
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paydays, sporting events, rainfall and holidays to predict crime "hot spots." 
Data-based decision making is transforming other fields. Medicine is using 
data analytics and computational modeling to predict who will get sick and 
insurance companies use it to determine high-risk clients. 

Research suggests that data-guided management is working for corpo¬ 
rate America. An MIT study of 179 companies found that those adopting 
data-driven decision making achieved productivity gains that were 5 to 6 
percent higher than other factors could explain (Brynjolfsson, Hitt, & Kim, 
2011 ). 

Using analytics, organizations are making intelligent decisions to better 
educate and serve society. Why is education so slow to respond to data- 
based decision making? In an arena that typically gathers an astonishing 
amount of data about its students, higher education has historically been 
inefficient in analyzing and using data for instructional and curricular 
development, allocating resources, or planning. However, now more than 
ever, the efficiency of educational institutions and the quality of academic 
programs are being challenged, and as a framework for making policy 
decisions and effecting organizational change, analytics could have a tre¬ 
mendous impact on higher education. 

DATA ANALYTICS IN HIGHER EDUCATION 

Data analytics is transformative, promoting not only organizational ef¬ 
ficiency and systemic change but also providing a model for educators to 
improve teaching and learning (Siemens & Long, 2011). We are in an era of 
data explosion. Because of the ubiquity of the Internet, mobile devices, and 
social networks, we have transitioned from ephemeral to captured, explicit 
data. Every online encounter, every Tweet or Facebook update, leaves a 
digital footprint, a personal data trail. Learning management systems (LMS), 
online courses, digital student records, and mobile devices capture streams 
of data about students that can be processed and analyzed. Data analytics 
has the potential to predict existing and upcoming challenges for institutions 
and students, drive decisions about financial resources and pedagogical 
systems, and offer a structure for improving academic outcomes, but it 
requires a different way of thinking about data and their value. 
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ACADEMIC ANALYTICS VS LEARNING ANALYTICS 

Academic analytics is the application of business intelligence in higher 
education. The focus of academic analytics is to improve decision making at 
the institutional level by producing actionable intelligence. As in business, it 
is the point where value is generated (Campbell, DeBlois, & Ob linger, 2007). 

Learning analytics differs from academic analytics in that it analyzes 
data about learners to understand and optimize learning and the learning 
environment. The focus is on the learning process, how students access and 
use content, and how they interact with instructors to create the learning 
space (Siemens & Long, 2011). It is an important tool for modifying and 
revising existing curricula and for planning courses and programs. From 
the intersection of technical, pedagogical, and social networks, including 
formal and informal education and work-life domains, learning analytics 
opens portals to new ways of teaching and learning. 

By analyzing student-performance behaviors, learning patterns, and 
interactions, institutions can make timely adjustments in course content, 
management, and support. In contrast to traditional evaluation methods, 
learning analytics provides information that can be used to make improve¬ 
ments in the same semester in which the data are collected (Siemens, 2011). 
The mechanism for learning analytics integrates technology, learning theory, 
and people in the organization to collect, process, and analyze data for 
applying and sharing knowledge. Analytics is more than evaluating what 
learners have done or predicting what they will do, it is about restructuring 
teaching and learning strategies to repurpose pedagogical paradigms and 
academic systems. Learning analytics is only as effective as the institutional 
capacity to accept and encourage a data-driven model of assessment. Ad¬ 
ministrative support and faculty buy-in are pre-requisites to creating an 
evidence-based and action-oriented culture (Norris et al., 2008). 

USING LEARNING ANALYTICS 

Many institutions across the country are applying learning analytics to pre¬ 
dict student success in terms of learning outcomes and to increase student 
retention and institutional accountability. Purdue University's SIGNALS 
and the University of California, Santa Barbara's Moondog projects are good 
examples of programs applying learning analytics to assess the chances 
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of students' success in a course, identify at-risk students and provide op¬ 
portunities for students to modify their academic behaviors and improve 
learning (Arnold, 2010; Zhang et al., 2008). 

LMS data allow faculty to evaluate the effectiveness of specific learning 
resources as correlated with student success. Research has shown that LMS 
student-tracking variables are pedagogically meaningful and statistically 
valid as predictors of final grades (Finnegan et al, 2010; Fritz, 2010; Mac- 
Fayden & Dawson, 2010). Other studies have reported that time on task and 
frequency of participation was statistically significant and therefore critical 
for online learning (Finnegan et al., 2010; Morris et al., 2001). 

Although many colleges and universities use analytics and LMS data 
to monitor student performance, faculty rarely use or understand learning 
analytics when making decisions about their own courses, student learning 
and retention, or instructional effectiveness. 

THE CASE STUDY 

Based on recommendations from the Higher Learning Commission, the 
provost and academic deans at a mid-size public Midwest university estab¬ 
lished an office for e-learning in fall 2010 to centralize online programming. 
A charter for distance learning, written and approved by the academic coun¬ 
cil, provided the framework for this new office. An associate dean and two 
full-time instructional designers were hired to develop institutional policies 
and protocols for implementing online courses and programs. The office, 
housed in the Division of Continuing Education, reported to the dean of 
the division. The success of the new enterprise was heavily dependent on 
showing that quality and rigor in course design and development could 
improve teaching effectiveness and increase student retention and course 
completion. It was also hoped that academic units and faculty would be 
encouraged to offer more online courses. 

Before the e-learning office was established, there was no formal process 
for designing or developing courses or reviewing courses for quality assur¬ 
ance before they were offered. Faculty put together online courses, of which 
there were approximately 400 using video lecture, without guidelines. Ac¬ 
cording to the university charter on distance learning, no new online course 
from this point on could be developed outside of the auspices of this office. 

The design-development process begins when the academic dean 
approves a course, after which faculty and designers flesh out learning 
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activities and assessments in line with learning objectives. A template for 
syllabus construction provides a guide for developing content, implement¬ 
ing teaching strategies, and articulating expectations for course completion. 
Appropriate technology and design elements are reviewed throughout 
the process. Course design, development, and implementation take five 
months from start to finish. 

By fall 2011, 13 new online courses were in place following these pro¬ 
cedures. Instructional designers from the e-learning office had met indi¬ 
vidually with faculty from Arts and Sciences, Social Work, and Education 
to design 10 undergraduate courses, including four in criminal justice, two 
in gender studies, two in history, one in Spanish, and one in social work. 
There were also three graduate courses, one in social work and two in 
education. Only one of the instructors had any online teaching experience. 
The courses would apply best practices and quality measures to sustain 
learning outcomes consistent with those in face-to-face classes and would 
be remarkably different from existing courses in terms of quality and rigor. 
Also, the services, support, and training that faculty received would be 
unmatched by former mechanisms. 

Feedback from faculty and students about their teaching and learn¬ 
ing experiences would provide an understanding of what influences and 
motivates learning in a virtual learning environment. In addition, LMS 
tracking data would be used to assess elements of course design, naviga¬ 
tion and management, delivery strategies, and learner participation. Based 
on critical analysis of real-time information extracted from the LMS and 
student mid-course surveys, areas of improvement and implications for 
instructional designers and course facilitators of the current as well as future 
virtual learning environments would be identified. 

Student mid-course survey 

Collecting the right information for the right reasons for the right persons 
at the right time was critical to the credibility of the efforts of the e-office. 
Instructors in the new courses distributed a 21-item mid-course student 
survey, which included open-ended questions, to gather data on student 
demographics, motivation, and satisfaction. Students completed and sub¬ 
mitted the survey within one week of distribution. 

Of the 36 percent of the students (N=186) who responded, three-quarters 
were female, 60 percent were 22 to 33 years of age, 68 percent were juniors 
or seniors, and 21 percent were graduate students. Forty-one percent held 
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full-time jobs; 29 percent worked half time or more, consistent for adult 
learners and what was expected for distance learners (Grant &Thornton, 
2007). 

The survey data were electronically processed on all courses collectively 
and for individual courses within a day of the submission deadline. One 
collective report and one individual report were sent to each instructor 
along with guidelines to help respond to student feedback. Instructors met 
with designers within a week of receiving results to review LMS data and 
to discuss implementing learning interventions, and make design changes 
and content revisions before the course ended. Learning interventions 
usually included web links to additional resources and adding or elimi¬ 
nating assignments. Design changes consisted of developing strategies to 
foster more discussion, providing easier access to assessments, and adding 
voiceover to PowerPoint presentations. Content revisions usually called for 
more clarity in presentation. The meeting also presented an opportunity 
to strengthen the instructor-designer relationship while learning how to 
apply information extracted from the LMS. 

Faculty mid-course survey 

At the same time that students were surveyed, instructors were also 
surveyed for feedback about their online teaching experience and com¬ 
petencies, i.e., engaging, motivating, and assessing students; creating and 
facilitating group activities and discussions; and using technology. They 
were also asked about services and support. Finally, instructors were asked 
to evaluate whether course design and development were conducive to 
learning and if implementation and delivery posed any barriers to student 
access or teaching effectiveness. 

Survey results 

More than 70 percent of students reported spending 3-6 hours per week in 
the course, that it was meeting academic needs, and that they would take 
another online course. In terms of course design, 94 percent were satisfied 
with the syllabus and learning objectives, and 91 percent found course navi¬ 
gation user-friendly and class policies clearly articulated. Over 84 percent 
found discussions valuable and instructor feedback prompt. Comparing 
online with face-to-face courses, 76 percent reported the online experience 
more effective and 65 percent felt connected to classmates. Open-ended 
questions confirmed that the majority of students were satisfied with their 
online learning experience, found it convenient and accessible, and learned 
that they had to manage their time to become effective online learners. 
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Seventy-four percent of the faculty considered themselves competent 
in motivating student participation, five percent said they were experts, 
and 21 percent described themselves as novices. Faculty (63 percent) felt 
competent in assessing students, with 21 percent claiming to be experts 
and 16 percent novice. With regard to group projects, 37 percent reported 
novice status, 29 percent said they were competent, and 26 percent of faculty 
never created groups. Over half said they were competent using technology 
and facilitating discussions, with about 25 percent claiming either novice 
or expert skills in both areas. Sixty-three percent of faculty said they used 
student tracking, 19 percent claimed expert status. However, only 50 percent 
considered themselves competent in using analytics, 32 percent reported 
being novices, 8 percent claimed expertise, and 10 percent admitted no 
competency. 

Other than the increased amount of time dedicated to online course 
development, most faculty (55 percent) found no differences in teaching on¬ 
line compared to the traditional format. Over 83 percent were satisfied with 
online course design and development services and ongoing support for 
implementation and delivery. Open-ended questions revealed that faculty 
missed the face-to-face interaction and that student time management and 
participation in discussions were crucial for successful course completion. 
Some faculty reported that they would re-think course design and re-do 
supporting materials for the next course offering. 

Applying learning analytics 

Overall, faculty perceptions of their online teaching skills were consistent 
with student feedback. However, their sense of competency with learning 
analytics and student tracking was not consistent with the perception of 
the e-learning office. We discovered that most faculty were unfamiliar with 
extracting LMS tracking data to assess student progress or identifying and 
offering learning interventions to struggling students. Faculty were also not 
making LMS data available to students to modify their academic behaviors, 
seek learning resources, or ask for academic support. 

Of the faculty who voluntarily met with instructional designers to 
discuss their course and options for enhancing the student learning ex¬ 
perience, all modified or revised elements in their current course rather 
than wait until the next course. One criminal-justice instructor completely 
revised content and changed formats by adding case studies and interactive 
simulations to the course. 
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We found that real-time information allowed faculty to intervene in 
ways that resulted in fewer student withdrawals or failing grades and 
more student retention (2.8 percent drop rate) compared to equivalent 
online courses that did not participate in the mid-course review and revi¬ 
sion process (3.0 percent drop rate) and to the face-to-face counterpart (4.7 
percent drop rate). 


RECOMMENDATIONS 

Over the long term, using analytics can be more efficient than traditional 
assessment or support measures for actionable improvements in teaching 
and learning because it provides a richer picture of student immediate 
practice. Faculty buy-in is imperative in creating assessment based on what 
students are learning. Faculty training and practice in tracking learning 
outcomes with an emphasis on results provide a measure of instructional 
effectiveness that is dynamic and invigorating. Our experience allows us 
to make the following recommendations: 

• Develop and provide guidelines to define clearly the 
data-driven process for course design and delivery us¬ 
ing linked data, e-learning theory, and best practices. 

• Establish mechanisms to collect, process, and analyze 
data on student performance during course delivery 
to identify students needing learning interventions for 
retention and success. 

• Establish a process for re-designing the online learning 
environments in real time by integrating learning ana¬ 
lytics, LMS data, and student/faculty feedback. 

• Provide ongoing services and support to instructors 
developing courses and teaching online. 

• Implement specific training in extracting and applying 
LMS student tracking data to improve teaching effec¬ 
tiveness and increase student learning. 

• Institutionalize online data-driven processes to provide 
continuing resources and support for program manage¬ 
ment and enhancement. 

These recommendations can become the foundation for policies and 
procedures related to offering and delivering online education and are also 
generalizable for broader use in other distance-learning environments. 


190 


CONTINUING HIGHER EDUCATION REVIEW, Vol. 76, 2012 


DATA-DRIVEN ONLINE COURSE DESIGN AND EFFECTIVE PRACTICES 


TIPS FOR IMPLEMENTING LEARNING ANALYTICS 

• Implementing learning analytics is not hard, but it can 
be messy when the control, distribution of results, and 
purpose are not properly communicated or supported 
from top administration. Implementation must take a 
value-layered approach. 

• Demonstrate analytics' uses and positive impact on 
organizational productivity and reputation. 

• Articulate the impact of analytics in higher education 
by communicating how institutions use data to make 
decisions about economic and pedagogical systems and 
to allocate resources for future planning. 

• Identify what is already being measured and determine 
what still needs to be measured and when. 

• Strengthen faculty and instructional designer relation¬ 
ships. 

• Use student-tracking data to assess instructional ef¬ 
fectiveness. 

• Create a community of practice to communicate and 
share analytics approaches and results openly across 
the institution to get buy-in from faculty and other 
stakeholders. 

CONCLUSION 

Analytics has a significant place in the future of higher education by guiding 
reform and system change. As this case study has shown, analytics can do 
more than evaluate what students have done and predict what they will 
do. Learning analytics can be transformative, altering existing pedagogical 
processes, research, data management, and policy-making. We were able to 
actualize real-time insight into student performance with targeted interven¬ 
tions, design changes, and content adjustments even as courses were taking 
place. Learners received feedback about their learning behaviors so that 
they could make in-course corrections for self-improvement. The increased 
collaborations among instructors, designers, and administrators improved 
the quality and value of the learning experience. 

This was new territory for us, but it is also new territory for education 
at a time when the way we teach and learn online is constantly evolving. 
Repurposing data collection and analysis to accommodate new modes of 
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teaching and learning and to make decisions about designing and deliv¬ 
ering online courses will be critical to student success and institutional 
productivity. 
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